feat: [iceberg] Native scan by serializing FileScanTasks to iceberg-rust #2528

mbutrovich · 2025-10-06T02:37:21Z

This PR introduces a new approach for integrating Apache Iceberg with Comet using iceberg-rust, enabling fully-native Iceberg table scans without requiring changes to upstream Iceberg Java code.

Rationale for this change

I was inspired by @RussellSpitzer's recent talk and wanted to revisit the abstraction layer at which Comet integrates with Iceberg.

Our current iceberg_compat approach requires code changes in Iceberg Java to integrate with Parquet reader instantiation, creating a tight coupling between Comet and Iceberg. This PR instead works at the FileScanTask layer after Iceberg's planning phase is complete. This enables fully-native Iceberg scans (similar to our native_datafusion scans) without any changes in upstream Iceberg Java code.

All catalog access and planning continues to happen through Spark's Iceberg integration (unchanged), but file reading is delegated to iceberg-rust, which provides better parallelism and integrates naturally with Comet's native execution engine.

What changes are included in this PR?

This implementation follows a similar pattern to CometNativeScanExec for regular Parquet files, but extracts and serializes Iceberg's FileScanTask objects:

Scala/JVM Side:

New CometIcebergNativeScanExec operator that replaces Spark's Iceberg BatchScanExec
Uses reflection to extract FileScanTask objects from Iceberg's planning output
Serializes tasks and catalog properties to protobuf for native execution

Native/Rust Side:

New IcebergScanExec operator that consumes serialized FileScanTask objects
Uses iceberg-rust's FileIO and ArrowReader to read data files
Leverages catalog properties to configure FileIO (credentials, regions, etc.)

How are these changes tested?

New CometIcebergNativeSuite with basic scenarios, but also a number of challenging situations from the Iceberg Java test suite
New CometFuzzIcebergSuite that we can adapt to Iceberg-specific logic
New IcebergReadFromS3Suite to test passing basic S3 credentials
Tested locally with Iceberg 1.5, 1.7, 1.10, CI tests 1.5.2, 1.8.1, and 1.10.0

Benefits over `iceberg_compat`

No upstream changes needed - No references to Comet needed in Iceberg Java anymore
Better parallelism - File reading happens at the same granularity as native_datafusion, not constrained by Iceberg Java's reader design
Simplified runtime - No separate DataFusion runtime; scans run in the same context as other operators
Better testing for iceberg-rust - I’ve already upstreamed several fixes for iceberg-rust’s ArrowReader
Multi-version support - Reflection approach is version agnostic

Current Limitations & Open Questions

We have a tracking issue on iceberg-rust for all of the changes from my branch we need to upstream to main ArrowReader enhancements for Apache DataFusion Comet iceberg-rust#1749
What is the behavior of a migrated Parquet table with INT96 values. I suspect need to pass the requested schema into ArrowReaderOptions to benefit from previous work in Arrow-rs Support different TimeUnits and timezones when reading Timestamps from INT96 arrow-rs#7285
Need to keep iceberg-rust in sync with Comet's DataFusion dependency (I had to bump my fork to DataFusion 50 for this PR, but iceberg-rust has now updated to DF 50)
Deprecation path for existing iceberg_compat code and its Iceberg Java entanglement

Related Work

Slides from the 10/9/25 Iceberg-Rust community call: iceberg-rust.pdf

codecov-commenter · 2025-10-06T02:54:32Z

Codecov Report

❌ Patch coverage is 67.74892% with 298 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.05%. Comparing base (f09f8af) to head (7911e4b).
⚠️ Report is 715 commits behind head on main.

Files with missing lines	Patch %	Lines
.../comet/serde/operator/CometIcebergNativeScan.scala	70.43%	82 Missing and 33 partials ⚠️
...n/scala/org/apache/comet/rules/CometScanRule.scala	44.51%	65 Missing and 21 partials ⚠️
...a/org/apache/comet/iceberg/IcebergReflection.scala	71.98%	62 Missing and 10 partials ⚠️
...e/spark/sql/comet/CometIcebergNativeScanExec.scala	81.05%	2 Missing and 16 partials ⚠️
...n/scala/org/apache/comet/rules/CometExecRule.scala	62.50%	2 Missing and 1 partial ⚠️
...rg/apache/spark/sql/comet/CometBatchScanExec.scala	70.00%	0 Missing and 3 partials ⚠️
...la/org/apache/comet/objectstore/NativeConfig.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2528      +/-   ##
============================================
+ Coverage     56.12%   59.05%   +2.92%     
- Complexity      976     1462     +486     
============================================
  Files           119      165      +46     
  Lines         11743    15060    +3317     
  Branches       2251     2504     +253     
============================================
+ Hits           6591     8893    +2302     
- Misses         4012     4899     +887     
- Partials       1140     1268     +128

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

comphead · 2025-10-06T15:22:35Z

It is promising!

# Conflicts: # native/Cargo.lock # spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala

…eberg version back to 1.8.1 after hitting known segfaults with old versions.

## Which issue does this PR close? - Part of #1749. ## What changes are included in this PR? - Change `ArrowReaderBuilder::new` to be `pub` instead of `pub(crate)`. ## Are these changes tested? - No new tests for this. Currently being used in DataFusion Comet: apache/datafusion-comet#2528

# Conflicts: # docs/source/user-guide/latest/configs.md # native/Cargo.lock # native/Cargo.toml # native/core/Cargo.toml

native/core/src/execution/operators/iceberg_scan.rs

spark/src/test/scala/org/apache/comet/CometFuzzIcebergSuite.scala

andygrove

Thanks @mbutrovich. I took a first pass through, and this is looking great! I'd like to get this merged in as soon as we can pin to a commit in the official iceberg-rust repo, and then continue the review once it is merged.

I'm going to review the serde parts more carefully later today.

andygrove

I don't see any reason not to go ahead and merge this as an experimental feature.

…llel test and setup workflows.

kazuyukitanimura

still looking

.github/workflows/iceberg_spark_test.yml

native/Cargo.toml

native/core/Cargo.toml

spark/pom.xml

spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala

…rkspace.

…<1.6.0

…due to type limitations in Iceberg 1.5.2.

kazuyukitanimura

I know this is merged, but one more comment

kazuyukitanimura · 2025-11-20T19:21:58Z

.github/workflows/iceberg_spark_test.yml

+        iceberg-version: [{short: '1.8', full: '1.8.1'}, {short: '1.9', full: '1.9.1'}, {short: '1.10', full: '1.10.0'}]
+        spark-version: [{short: '3.4', full: '3.4.3'}, {short: '3.5', full: '3.5.7'}]


The profile now says to use Iceberg 1.5 with Spark 3.4, but we do not have 1.5 here. Not sure if it causes problems...

Here's what we currently test with this PR:

3.4 3.5 4.0

1.5.2 CometIcebergNativeSuite CometFuzzIcebergSuite IcebergReadFromS3Suite (not run in CI due to MinIO container)

1.8.1 Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests CometIcebergNativeSuite CometFuzzIcebergSuite IcebergReadFromS3Suite (not run in CI due to MinIO container)

1.9.1 Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests

1.10 Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests CometIcebergNativeSuite CometFuzzIcebergSuite IcebergReadFromS3Suite (not run in CI due to MinIO container)

I leaned on newer versions for the Iceberg tests because as best as I could tell, never versions are a superset of the older versions. For the Comet-native tests we are running 1.5.2.

We should have a discussion of what we want to run long term, because right now tagging a PR [iceberg] makes CI take hours and causes so many parallel Iceberg suites that we start getting network timeouts (likely due to throttling).

mbutrovich added 3 commits October 5, 2025 21:53

CometNativeIcebergScan with iceberg-rust using FileScanTasks.

cded0ad

Clean up tests a little.

4f3004b

Remove old comment.

4afec43

mbutrovich added 6 commits October 6, 2025 06:58

Fix machete and missing suite CI failures.

fc97ce9

Fix unused variables.

cca4911

Spark 4.0 needs Iceberg 1.10, let's see if that works in CI.

93f466d

Remove errant println.

970b692

Remove old path() code path.

c44973b

Update old comment.

0f83fd4

mbutrovich added 2 commits October 6, 2025 11:49

Iceberg 1.5.x compatible reflection. Use 1.5.2 for Spark 3.4 and 3.5.

6cbbd09

Fix scalastyle issues.

6966a12

mbutrovich changed the title ~~feat: Iceberg scan based serializing FileScanTasks to iceberg-rust~~ feat: [iceberg] Scan based serializing FileScanTasks to iceberg-rust Oct 6, 2025

mbutrovich force-pushed the iceberg-rust branch from 227332c to 6966a12 Compare October 6, 2025 20:03

mbutrovich changed the title ~~feat: [iceberg] Scan based serializing FileScanTasks to iceberg-rust~~ feat: Iceberg scan based serializing FileScanTasks to iceberg-rust Oct 6, 2025

mbutrovich added 7 commits October 7, 2025 13:03

Merge branch 'main' into iceberg-rust

1153d71

# Conflicts: # native/Cargo.lock # spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala

Remove unused import.

a0f4d63

Clean up docs a bit.

a9cebfd

Refactor and cleanup.

6b2175a

Refactor and cleanup.

3618407

Add IcebergFileStream based on DataFusion, add benchmark. Bump the Ic…

8091a81

…eberg version back to 1.8.1 after hitting known segfaults with old versions.

Fix CometReadBenchmark.

880599e

This was referenced Oct 15, 2025

feat(reader): Make ArrowReaderBuilder::new public apache/iceberg-rust#1748

Merged

ArrowReader enhancements for Apache DataFusion Comet apache/iceberg-rust#1749

Open

mbutrovich added 4 commits October 16, 2025 16:04

Merge branch 'main' into iceberg-rust

5127e1c

# Conflicts: # docs/source/user-guide/latest/configs.md # native/Cargo.lock # native/Cargo.toml # native/core/Cargo.toml

Fixes after bringing in upstream/main.

878c971

Basic complex type support.

e66799e

CometFuzzIceberg stuff.

4f2f3b8

Update Spark diffs for new arg in CometBatchScanExec.

224b3e7

andygrove self-requested a review November 17, 2025 15:21

andygrove reviewed Nov 17, 2025

View reviewed changes

native/core/src/execution/operators/iceberg_scan.rs Show resolved Hide resolved

andygrove reviewed Nov 17, 2025

View reviewed changes

spark/src/test/scala/org/apache/comet/CometFuzzIcebergSuite.scala Show resolved Hide resolved

andygrove reviewed Nov 17, 2025

View reviewed changes

spark/src/test/scala/org/apache/comet/CometFuzzIcebergSuite.scala Show resolved Hide resolved

andygrove reviewed Nov 17, 2025

View reviewed changes

mbutrovich added 3 commits November 17, 2025 12:00

Merge branch 'main' into iceberg-rust

030c530

Merge branch 'main' into iceberg-rust

974fe94

Switch to upstream iceberg-rust.

7c0a99b

andygrove approved these changes Nov 18, 2025

View reviewed changes

mbutrovich added 9 commits November 19, 2025 10:48

Merge branch 'main' into iceberg-rust

4d3ffe5

Merge branch 'main' into iceberg-rust

6a528fc

Fix q79 plans? Not sure why this is needed.

8be08b8

Move iceberg-rust-related diffs to their own folder, and add new para…

67a0cb6

…llel test and setup workflows.

Merge branch 'main' into iceberg-rust

9ecd93a

Merge branch 'main' into iceberg-rust

68652d4

Update comment in pom file.

070590a

Update iceberg-rust workflow.

46c507e

Fix existing Iceberg integration.

773eded

kazuyukitanimura reviewed Nov 19, 2025

View reviewed changes

.github/workflows/iceberg_spark_test.yml Outdated Show resolved Hide resolved

native/Cargo.toml Show resolved Hide resolved

native/core/Cargo.toml Outdated Show resolved Hide resolved

kazuyukitanimura reviewed Nov 19, 2025

View reviewed changes

spark/pom.xml Show resolved Hide resolved

spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala Outdated Show resolved Hide resolved

spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala Outdated Show resolved Hide resolved

mbutrovich added 7 commits November 19, 2025 18:16

Fix match arms for old Iceberg integration.

cef5390

Move datafusion-datasource up to top cargo.toml, and set core's to wo…

46f07d2

…rkspace.

Add Spark 3.4 to Iceberg Java workflows for Iceberg-Rust code path.

cffc791

Iceberg 1.5.2 for Spark 3.4.

80b5fdc

Omit incompatible types from CometFuzzIcebergBase schema for Iceberg …

a280467

…<1.6.0

Remove test print.

c6c3021

Adjust schema filtering logic in fuzz test "order by random columns" …

7911e4b

…due to type limitations in Iceberg 1.5.2.

mbutrovich merged commit 937cacd into apache:main Nov 20, 2025
259 of 261 checks passed

kazuyukitanimura reviewed Nov 20, 2025

View reviewed changes

		iceberg-version: [{short: '1.8', full: '1.8.1'}, {short: '1.9', full: '1.9.1'}, {short: '1.10', full: '1.10.0'}]
		spark-version: [{short: '3.4', full: '3.4.3'}, {short: '3.5', full: '3.5.7'}]

	3.4	3.5	4.0
1.5.2	CometIcebergNativeSuite CometFuzzIcebergSuite IcebergReadFromS3Suite (not run in CI due to MinIO container)
1.8.1	Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests	Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests CometIcebergNativeSuite CometFuzzIcebergSuite IcebergReadFromS3Suite (not run in CI due to MinIO container)
1.9.1	Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests	Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests
1.10	Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests	Iceberg Spark Tests Iceberg Spark Extensions Tests Iceberg Spark Runtime Tests	CometIcebergNativeSuite CometFuzzIcebergSuite IcebergReadFromS3Suite (not run in CI due to MinIO container)

feat: [iceberg] Native scan by serializing FileScanTasks to iceberg-rust #2528

feat: [iceberg] Native scan by serializing FileScanTasks to iceberg-rust #2528

Uh oh!

Conversation

mbutrovich commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Scala/JVM Side:

Native/Rust Side:

How are these changes tested?

Benefits over iceberg_compat

Current Limitations & Open Questions

Related Work

Uh oh!

codecov-commenter commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

comphead commented Oct 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

kazuyukitanimura left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kazuyukitanimura left a comment

Choose a reason for hiding this comment

Uh oh!

kazuyukitanimura Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

mbutrovich Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mbutrovich commented Oct 6, 2025 •

edited

Loading

Benefits over `iceberg_compat`

codecov-commenter commented Oct 6, 2025 •

edited

Loading